Extensible Tokens

Problem description

A scanner recognizes various tokens and performs various actions on them.  In this project, design the implementation of tokens such that they can be augmented without changing the existing code.  In other words, write a reuseable token representation so that other people can extend it without having to make any change in the base code.

For example, one may start with tokens such as Identifier and Number.  Each type of tokens have its own operations such as GetType or Print.  Then someone may later want to support a new type of tokens, for example, BinaryInt such as x10001. In addition, someone else may want to add a new function to process all tokens, for instance, Count, which counts the number of tokens of each type.  The question is how you can design the representation of tokens so that later additions can be easily added.  The specific requirements are as follows.

Project organization

Recommended reading

Thinking in C++. Eckel, Prentice Hall, 1995.  (Use it as a C++ reference book, it contains clear explanations and concise examples).  I have a copy of the book that you can borrow for a short period of time (so that other groups can use it as well).

Design Patterns: Elements of Reusable Object-Oriented Software, Gamma, Helm, Johnson, and Vlissides, Addision-Wesley, 1995.  Read the section on Visitor pattern, pp. 331-344.  I have a copy of the book that you can borrow for a short period of time (so that other groups can use it as well).

Synthesizing Object-Oriented and Functional Design to Promote Re-Use. Krishnamurthi, Felleisen and Friedman, Technical report TR 98-299, Rice University, 1998.